Bonus or Not? Learn to Reward in Crowdsourcing
نویسندگان
چکیده
Recent work has shown that the quality of work produced in a crowdsourcing working session can be influenced by the presence of performancecontingent financial incentives, such as bonuses for exceptional performance, in the session. We take an algorithmic approach to decide when to offer bonuses in a working session to improve the overall utility that a requester derives from the session. Specifically, we propose and train an inputoutput hidden Markov model to learn the impact of bonuses on work quality and then use this model to dynamically decide whether to offer a bonus on each task in a working session to maximize a requester’s utility. Experiments on Amazon Mechanical Turk show that our approach leads to higher utility for the requester than fixed and random bonus schemes do. Simulations on synthesized data sets further demonstrate the robustness of our approach against different worker population and worker behavior in improving requester utility.
منابع مشابه
Deep Learning for Reward Design to Improve Monte Carlo Tree Search in ATARI Games
Monte Carlo Tree Search (MCTS) methods have proven powerful in planning for sequential decision-making problems such as Go and video games, but their performance can be poor when the planning depth and sampling trajectories are limited or when the rewards are sparse. We present an adaptation of PGRD (policy-gradient for rewarddesign) for learning a reward-bonus function to improve UCT (a MCTS a...
متن کاملTSEB: More Efficient Thompson Sampling for Policy Learning
In model-based solution approaches to the problem of learning in an unknown environment, exploring to learn the model parameters takes a toll on the regret. The optimal performance with respect to regret or PAC bounds is achievable, if the algorithm exploits with respect to reward or explores with respect to the model parameters, respectively. In this paper, we propose TSEB, a Thompson Sampling...
متن کاملLinking strategy, performance, and pay.
The appraisal portion of the strategic plan has long been a problem. How do you reward the achievement and performance of individuals as they operate the organization's strategic plan? How do you use the performance appraisal area to motivate the management team to achieve the objective in the strategic plan? How do you provide incentive for your people to stay with your organization? How do yo...
متن کاملEarly and late consolidation and reconsolidation of memory in the prelimbic cortex
Rats can learn to forage among olfactory cues to associate one with reward in only 3 massed trials. The learning is achieved in less than 10 min and results in a memory trace lasting at least 1wk week. To study the neuro-anatomical circuits involved in the memory formation we used immunoreactivity to the immediate early gene c-fos as a marker for neuronal activity induced by the learning. The p...
متن کاملPerform Three Data Mining Tasks with Crowdsourcing Process
For data mining studies, because of the complexity of doing feature selection process in tasks by hand, we need to send some of labeling to the workers with crowdsourcing activities. The process of outsourcing data mining tasks to users is often handled by software systems without enough knowledge of the age or geography of the users' residence. Uncertainty about the performance of virtual user...
متن کامل